Hierarchical Multi-Attention Transfer for Knowledge Distillation

Knowledge distillation (KD) is a powerful and widely applicable technique for the compression of deep learning models. The main idea knowledge to transfer from large teacher model small student model, where attention mechanism has been intensively explored in regard its great flexibility managing different teacher-student architectures. However, existing attention-based methods usually similar intermediate layers neural networks, leaving hierarchical structure representation poorly investigated distillation. In this paper, we propose multi-attention framework (HMAT), types are utilized at levels Specifically, position-based channel-based characterize low-level high-level feature representations respectively, activation-based both mid-level representations. Extensive experiments on three popular visual recognition tasks, image classification, retrieval, object detection, demonstrate that proposed or HMAT significantly outperforms recent state-of-the-art KD methods.